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ABSTRACT 

This document reports on the 1987 external review of 
the nine Regional Educational Laboratories which evaluated lab 
performance during the first 18-20 months of their contracts and 
evaluated their plans for the remaining 3 years of the contracts. 
Observations about lab plans are presented on a program-wide basis. 
It is reported that the 3-5 year plans developed by the labs leave 
questions for the review panel about the overall clarity and vision 
of the labs' missions. This appears to led to uncertainty in the area 
of programmatic decision-making. The way in which labs set priorities 
was seen as not always clear. Some labs have experienced a profound 
shift of identity and purpose from being developer and provider of 
research and development to that of a linking agency, working 
primarily with and through other educational service agencies. The 
following questions are discussed: (1) Are the labs to be pro-active 
or reactive within their regions? (2) To what extent should regional 
labs also have some national identity? and (3) Are the intended 
degree and benefits of collaboration among the labs being achieved? 
Recommendations are made regarding the future of the program. (JD) 
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October 5, 1987 

Dr. Chester E. Finn, Jr. 
Assistant Secretary and 

Counselor to the Secretar- 
ies. Department of Education 
400 Maryland Avenue, S.W. 
Washington, D C. 20202 

Dear Checker: 

The report of the Laboratory Review Panel on the review of Regional Educational 
Labs is forwarded to you. 

The Panel is pleased to have had the opportunity to comment on the review As 
noted in the report, we believe the review was conducted verv competentl\ and 
that the results will be beneficial to the labs and to OERI. 

We have focused, for the most part, on program-wide issues in our report We have 
utilized the opportunity to draw upon all of the reports on individual labs to help 
identify issues for your consideration. 

You will see, however, that we have some specific recommendations and 
observations n urding three individual labs. In our judgement, these matters 
warrant specia attention. We hope you will consider these recommendations, as 
well as the others, and advise us at an appropriate time of any actions taken with 
respect to these matters. 

I would be pleased to speak with you to discuss this report, as would 
Joy Frechtlmg and Garry McDaniels, other Panel members who are in this area 

The Panel plans to meet three times per year hereafter on a regular basis Our 
agenda from here forward will concentrate on monitoring the results of the revieu. 
discussion of forthcoming evaluation activities, and providing input to po!ic\ 
regarding the future of the program. 

If you have suggestions about how the Panel may be of further assistance tc vou. 
please let me know. 



Sincerely^ 




Christopher'?. Cross 
Chairman 

Laboratory Review Panel 



CTC/ab 
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REPORT OF THE LABORATORY REVIEW PANEL 
ON THE 198? EXTERNAL REVIEW OF LABORATORIES 



ERIC 



I. Observations About the Revlev Process - 

In the summer of 1987, OERI conducted an external peer review of the 
nine regional educational labs. The purpose of the review was to 
evaluate lab performance during the first 18-20 months of their 
contracts and also to evaluate their plans for the remaining three years 
of the contracts. 

A team of external reviewers, accompanied by the cognizant OERI 
institutional liaison (IL), visited each lab for two and one-half days. 
The teams had been trained in Washington for two days prior to the 
on-site visits. The teams read the labs' 3-5 year plans and other 
documents necessary for their work. Standardized evaluation criteria had 
been developed for review of both lab performance and plans. 

While on-site, the teams met with lab board members, management and 
staff. The teams also contacted selected lab clients and constituents by 
telephone to determine their perceptions of the labs' performance. The 
review was conducted within relatively short timelines and with 
relatively limited resources. 

The panel has compiled information about the review from several 
sources: (a) reports from review. teams and OERI institutional liaisons 
(IL s); (b) meetings with the review teac leaders, IL's and lab 
executive directors; (c) written comments from the executive director 
concerning their reviews; (d) visits to a lab by most panel members 
while a site visit was in progress; and (e) review of lab 3-5 year plans 
and needs assessments. In addition, the panel chair briefly observed the 
review teams' training and spoke to the reviewers. 

Before proceeding with detailed comments on the review, the panel has 
some preliminary observations to make about its overall nature and 
findings. The OERI design generated much factual information about 
specific aspects of lab performance and plans. Reviewers absorbed a 
great deal of written and oral information about the labs and their 
reports are very factual. 

The review generated relatively little information about external 
considerations in the labs' environments. For example, while information 
is available about constituents' and clients' perceptions of the labs, 
there is not documented evidence about labs' effects in the regions. Nor 
did the review address such a question as: "Is this the optimal 
arrangement of service improvement organizations in the regions'" OERI 
staff did not see the review as capable of addressing such broader 
Issues. It plans to address such Issues in other parts of its lab 
tvaluation plan. The panel concurs with the OERI position in this 
regard. 

Detailed comments on the review process follow. 
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The review was exc eptionally veil organized and implemented. Ir. 
particular: 

- The reviewers were broadly representative of lab constituent groups. 
They were selected as a result of a wide search process which included 
requests for nominations from a larger number of professional 
organizations and other sources. The reviewers therefore brought a fre-- 1 - 
view to the evaluation of labs. The panel regards this outreach 
positively. 

- OERI spent considerable time and effort in preparing and training the 
reviewers. These preparations apparently provided teams with a common 
understanding of the purposes and criteria for the review. Room for 
appropriate discretion was nonetheless left for review teams regarding 
the details of individual reviews. Reviewers were motivated and 
industrious while on-site. 

- The reviewers were sensitive to the issues faced in the labs. Their 
backgrounds provided ther with relevant perspectives with which to do 
their job. 

- Communication from OERI to the labs about the review was good. 
Information about procedures, criteria and schedules was communicated to 
the labs in advance. This apparently facilitated understanding and 
acceptance of the review by the labs. 

- The labs were very open to the review process. They responded to 
questions and provided information freely and, in general, treated the 
review as a learning opportunity. 

- The OERI institutional liaisons (IL's) played a key role in tht 
review. They provided historical and Departmental perspectives r out the 
labs which would otherwise not have been available to the panel. Thev 
also provided en additional source of information on numerous key 
issues. This information helped the panel "triangulate" data soi'-ces and 
perspectives in its work. 

Vith-hlndslght, some things might be done differently. We recommend the 
following changes In any future reviews; 

- Reviewers be given exemplars of critical characteristics of "good 
labs" against which to make comparisons. 

- Each team should review more than one lab. The teams in this review 
did not have such cross-cutting assignments or membership. This 
prevented reviewers from having aore than one basis for making judgments 
about lab performance and plans. In general, the panel believes there is 
a tendency for reviews of this type to produce positive results. 
Institutions being reviewed find it helpful to have a person to talk 
with, explain one's programs, etc. This does not mean the results fron: 
this review are to be disbelieved, but the tendency for positive 
findings to result from sjch a process should be kept in mind. 
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- The panel received rc'-tively little information about the qualit- o 
impact of lab products. The panel recommends further information about 
this topic be developed in the future. If not done through such a 
review, it should be done through other means. 

- Consideration should be given to paying reviewers. In this case, the 
teams did a tremendous amount of work, even going beyond the number of 
days they nominally agreed to serve. However, some of them had 
reservations about non-payment and it is unlikely that many of them 
would volunteer to perform the same service on an unpaid basis again. 
This would impair OERI's ability to obtain consistency in review teairs 
over time. (Further, non-payment would have been a severe barrier in 
this case to any attempt by OERI to have teams review more than one 
lab. ) 

Taking in to account all the observations and caveats above, the panel 
considers this review exemplary and credible in light of the time and 
resources available. 



II. Observations About the 3-5 Year Plans 

The panel wishes to frame its observations about lab plans drawn froo 
this review on a program-wide basis. It believes that such a perspective 
can best complement the laboratory-specific orientation of the external 
review teams. 

The panel does not believe it has sufficient information to qualify 
review tean's findings about individual laboratories' performance. It 
believes more information about lab impact from the field is desirable. 
The panel will say, however, that it has no evidence of gross 
discrepancies between labs' stated commitments and pe ormance to date. 

This section will therefore comment or program-wide Issues and 
considerations which have been raised through the panel's discussions of 
and participation in the review. The panel believes these are generallv 
unresolved matters and warrant the Assistant Secretary's attention. 

The panel thinks regards the present level of effort being expended by 
OERI and the developing nature of its relationships with the labs 
postively. The panel is not under any illusions that the relationships 
are trouble-free. But it believes (particularly its members who are 
knowledgeable about prior administrative arrangements within NIE/OERI) 
that the assignment of Individual OERI staff for major parts of their 
time as Institutional liairons, together with focused OERI management 
v attention to the program, is praiseworthy. 

Incorporation in the program of the Indirect services strategy (working 
with and through intermediaries) is a significant development in the 
labs history. The strategy has affected the kind, of work the labs 
conduct. It also has implications for the appropriate way to evaluate 
labs work. The panel does not believe the full implications of this 

ERIC 



ftTbelc" UnderSt °° d " M °" ^tailed counts on this topic are 

Unresolved Issues and Considerations 

The 3-5 year plans deve loped bv the labs leave questions for the panel 
about the overall clarity a nd vision of t he labs' missions- On Jt„ m „ 
trie lab 3-5 year plans and accompanying needs assessments were abn U t *00 
pages per institution. This often made it difficult for the panel to *et 
a clear picture of what a lab 1. doing. Stylistically, executive 
summaries of the plans Bight help. Alternatively, the entire plans could 
•imply be written more succinctly. 

In particular, the panel frequently found it hard to get a clear idea cf 
what overall sense of mission drove the lab plans. For example, whether 
a lab sought to better enable its constituencies to handle change on " 
their own. to "gap fill" with needed services, or to act a f a 
disseminator of information within its region (or some combination of 
these) was not always clear. Ironically, the extensive amount of 
verbiage in the plans did not help with this problem. 

The panel believes one by-product of such lack of clarity is that it is 
difficult to. set reasonable expectations for lab performance. As a 
corollary observation, the panel believes the labs' self assessments may 
be relatively stronger in assessing specific activity-level performance' 
and weaker in assessing program-vide or institutional performance, 
especially impact. 

There is an additicral observation by the review teams which the panel 

£ JIT! I*? b ! r ? latBd ? J'? P er «P tien ° f l«k of clarity and vision 
in the labs missions and their -ole in the school improvement process. 
That observation was sometimes ambiguous internal guidelines in use by 

wJen r° T d " , ™ ll,1 ; £ when t0 off " °' "rvices to constituents 

when requested, and when tc stop services once at.rted. Lack of a larger 

All of L l8bor V tor y 8 » i8sion ~y 1" P«t lead to uncertainty in this 
area of programaiati c decision-making. 

The way in which labs set priorities is not »iw, vc cles r- The panel 
?°5*5 that extensive needs assessments have been done to~help guide the 

JjraineV^M * ^W" 0 " ,Xp,nded 8nd -ount.M data 
relL^.M J V™ 8 " hlgh * But the P« el frequently did not see the 
relationships between needs assessment data and the choice of programs 
or strategies made by the labs. programs 

111?*: the r dlfficult y * n •■tablishing auch relationahips lay with the 
complexity of the labs document at ion, discussed above. In part, the 

tl » ESS S V J th ?rj llc8t8d n " u " of the need. JS2i£ data 
themaelvcs. The data would in many cases support numerous strategies anc 
programs, more in fact, than the lab could ever hope to respond to. In 

«S. C ;!r!' I? ".sons underlying the actual choice, made about whom to 
•trvt vere not always cltar. 
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Stated differently, the panel believes there may be patterns of *ervice 
allocation to different populations and areas of their regions made by 
the labs which can not be predicted by the needs assessment data and 
which may not reflect any other stated premises. The panel recommends 
more succinct information be presented in the future regarding reasons 
for adoption of lab strategies and allocation of services. 

In a similar vein, some pjans contain commendable statements of regional 
context considerations, in part, we understand, in response to OERI 
requests. But it is not always clear that the actual choice of programs 
undertaken by a lab has any direct relationship to the context 
considerations described. One lab, for example, noted the severe 
economic considerations in its region, but its programs did not deal 
directly with finance issues, or how schools in its region might deal 
with budgetary crises. Greater synergy between contextual considerations 
outlined in the plans and the plans themselves is desirable. 

The implications of the indirect service ("with and through 0 ) strategies 
need furt her examination- The panel vigorously sought information about 
hov indirect service strategies are working in the labs. These 
strategies are clearly a major aspect of the labs 1 programming. The 
panel believes the labs have established networks where they did not 
exist before as a result of the strategics. It also believes the labs 
serve clients through existing networks as well. But the impact of 
indirect service strategies is felt not only vith respect to work 
performed specifically under Task Two of the contracts (where the 
provision if contained), but also with regard to how a lab perceives of 
itself as ? institution. In the latter regard, some labs have 
experienced a profound shift of identity and purpose from being a 
developer and provider of R&D to that of a linking agency, working 
primarily with and through other educational service agencies. 

The following specific aspects of the indirect service strategies are 
highlighted for consideration. 

- There needs to be a better understanding of the range of partners vith 
whom labs might work and which choices are most efficacious. 

- Indirect service strategies have different implications in different 
type regions (e.g., for the Far Vest Lab's, where there are many 
intermediaries and the Appalachia Lab's, where there are relatively 
few). 

- The lab RFP states acceptable conditions under which labs may work 
directly at the local level. In general, these guidelines appear to have 
been followed. In most cases, for examples labs do not appear to define 
building level personnel as a primary type of client. There are 
individual cases, however, where a lab is working directly at the local 
level in which it is not clear that it has a mandate to do so. Some 
clarification of these guidelines as they relate to the 3-5 year plans 
may be desirable. 

- The strategies may subject labs to uncontrollable influences. For 
0 example, the Governor of California recently cancelled a proaram of 
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technical assistance centers with which the Far West Laboratory was 
working as part of its indirect services. This development has required 
reprograming by the lab, 

- The 1985 recompetition resulted in a transition of labs fron 
institutions which conducted some significant R&D on their own, to ones 
providing assistance services, primarily in partnerships with others. 
While the change has clear benefits, one cost is the loss of 
practitioner-oriented R&D that labs used to conduct. In part, this 
transition makes the choice of the R&D that labs Incorporate in their 
services more critical. Based on knowledge presently available to it, 
the panel is not sure that there is a sufficient locus of 
practitioner-oriented research emanating from other sources which the 
labs may draw upon. 

• As indicated earlier, the panel believes more information about lab 
impact is desirable. But while the ultimate goal of labs is school and 
classroom improvement, it isn't clear that looking for such outcomes is 
the appropriate criterion for a lab using indirect service strategies, 
except in given situations where there is a discernible linkage between 
services delivered by the lab and the ultimate school beneficiaries. 
Such situations are probably the exception rather then the rule. Further 
attention to this issue is warranted. 

Are the labs to be pro-aztive or reactive within their regions? The 
panel perceives an issue vhich appears to be unresolved regarding the 
labs' role in their regions: the degree to which they are to be 
pro-active in identifying areas for involvement and taking leadership in 
those areas, and the degree to which they should be 
"constituent-driver". 

This issue reflects facets of the other issues vhich have preceded it in 
this section of the report, i.e., whether the labs have an overall 
vision of their mission, whether their needs assessments are adequate 
and their programs related to them, and the implications of the indirect 
service strategies for the institutions. But new considerations are 
raised as well. 

On the one hand, when a lab has a clear sense of constituent needs and 
follows them, it will presumably play a useful role in its region. But 
Its programs may be changing and unrecognizable from one year to the 
next. On the other hand. If a lab stakes out leadership in an area, it 
risks both political backlash and changing priorities which may render 
its capabilities and agenda obsolete. 

In this regard, the panel notes that some labs have identified areas In 
vhich they are particularly strong. That la good; however, a team in one 
case haa reported that the lab had a tendency to diagnose or interpret 
client problems in terms of a school improvement area in which it had 
particular expertise and then frame its services in that area. 

Aa indicated, the pro-active/reactive Issue is but one aspect of others 
raiaed here, perhaps moat atrongly, that of what is the larger vision 
labs have of their mission. The panel believes the laba must strike a 
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balance between being pro-active and reactive. It is not sure such a 
balance is understood or has been achieved among the labs. 

To what degree shou ld regional labs also have some national identity? 
The panel has inquired about the extent to which labs perceive 
themselves as being oriented entirely to their regions, or whether they 
have, or ahould have, some national identity or outlook as well. 

The panel's perception is that the labs are very strongly oriented to 
their regions. This is a strength. On the other hand, there are some 
legitimate roles outside the region which labs might become involved 
with. 



Appropriate national visibility and orientation can be achieved through 
national recruitment and staffing of labs' key positions. Participation 
in appropriate national professional activities is also appropriate. 
Laboratories do these things. One important national role might be tc 
exert leadership in one or more areas of school improvement. This might 
be done through original collaborative arrangements with parties 
outside, as well as within, the labs' regions. Collaboration will be 
discussed below. 



One aspect of the labs' role which the panel finds troubling is an 
apparent lack of concerted effort by either OERI or the institutions to 
develcp a general plan for fostering collaboration with other R&D 
resources funded by the Department of Education. The panel has 
identified numerous assistance activities funded by the Department, with 
whoir the labs might collaborate. These are shown on Table 1 (following 
page). 

The panel believes more collaboration can be achieved between labs and 
other Departmentally-funded resources and strongly urges that 
appropriate action be taker, to bring about such collaboration. 

Collaboration among labs - The panel questions whether the intended 
degree and benefits of collaboration among the labs are being achieved. 
The 1985 RFP stipulated that labs should "work in collaboration with 
centers and with other labs on regional and national educational 
problems". Laboratories are to allocate ten percent of their budgets to 
collaboration under Task Five to participate in activities that "address 
more than one region or are nationwide in scope". Specific Task Five 
actlvitlea may Include: 

•xchange of information on R&D needs and practices through 
meetings, newsletters or electronic networks; 

development of resources for improvement, e.g., syntheses, training 
modules, workshop designs; 

engagement in collaborative improvement efforts across regions; 

assisting OERI in understanding needs of educational practitioners 
in regions and nationwide; 
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Table 1 

OTHER ASSISTANCE RESOURCES FUNDED 
B7 THE U.S. DEPARTMENT OF EDUCATION 
WITH WHOM REGIONAL LABORATORIES MIGHT COLLABORATE 



Offlce/Activitv 



Funding 

($ in millions) 



Educational Research and Improvement (OERI) 

Educational Research Centers (18) i 7 . 8 
ERIC Clearinghouses (16) 



4.8 



4.8 



National Diffusion Network (NDN) 
State Facilitators (53) 

Leadership in Educational Administration 

Development (LEAD) Program Centers (51) 7.1 



Elementary and Secondary Education (OESE) 

Chapter I Technical Assistance Centers (4) 3.6 



Indian Education- Regional Resource 
Centers (5) 

Drug Free Schools Centers (5) 



Special Education and Rehabilitative Services 

(OSERS) 

Rtgionsl Resource Centers (5) 



2.2 
8.8 



2.2 



(OBEKLA** Educ * ti0n * nd Kinority Language Affairs 

Mul.ti-Functionsl Resource Centers (16) 10 ,o 

Evslustion Assistsnce Centers (2) 0 7 

Bilingusl Education Clesringhouse l 0 
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Table 1 (continued) 



OTHER ASSISTANCE RESOURCES FUNDED 
BY THE U.S. DEPARTMENT OF EDUCATION 
WITH WHOM REGIONAL LABORATORIES MIGHT COLLABORATE 



Civil Rights (OCR) 

Desegregation Assistance Centers (10) 8.2 

Vocational and Adult Education (OVAE) 

National Center for Research on Vocational 

Education (NCRVE) . 6,0 

Community Coordination Centers (6) 0.8 

Total Funding 78, o 



The regional offices of the Department (Secretary's Regional 
Representatives) are also resources with whom the labs might work. 
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e) cooperating in work in national research, development, 

dissemination and improvement problems jointly identified bv tne 
lab and OERI. 

A central coordinating group was described in the RFP that would be 
convened by OERI in consultation with the labs. It would determine 
guideline!, procedures and priorities and would involvr assignments to 
cross-laboratory task forces for planning and implementation. An annual 
meeting of lab governing board chairs, executive directors and OERI was 
also envisioned. 

Nine "theme areas" emerged from lab plans in FY ? 86. Three of these were 
administrative in nature, including electronic networking and 
evaluation. The electronic networking has occurred as envisioned and 
some effective collaboration jn substantive themes, e.g., higher order 
thinking skills, the urban education network, state policy and rural 
education. Collaboration in other areas, however, has lagged and except 
for two of the above areas, content-based product? have not been 
developed as yet. 

OERI staff have reported severr 1 difficulties with Task Five 
implementation to date. One concerns laboratory leedership for specific 
themes. There is a lack of balance and consistency among labs ii taking 
leadership for themes. Some labs have taken the lead in more t v <*n one 
area and at least one has not taken the lead in any. Lead IVos cannot 
force other labs to cooperate and lead lab leadership is very fragile; 
it sometimes does not emerge and is usually highly person-dependent. 

It has been difficult to identify the overall dimensions of Task Five 
because the work in the labs often overlaps that in other tasks. 

OERI staff have made the following recommendations regardirg the 
implementation of Task Five: 

1) The notion of the "lead lab" should be re-evaluated in 
cooperation with the labs. The number of "lead" areas per 
lab may need to be limited. 

2) A more equitable distribution of work among labs be 
determined in collaboration with them. 

3) OERI should take a more active and collaborative role in 
implementing Task Five activities, ideally by having each member 
of the CERI Laboratory team serve as a liason/f acilitator for a 
theme area. 

4) The institutional liaisons would have responsibility for generating 
collaborative efforts across OERI and the Department. 

5) Products developed under Task Five should be disseminated 
nationally through a variety of educational agencies. 

6) Meetings among OERI staff and lab staff should continue as 
necessary to reach the#e objectives. 

13 
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The panel endorses the general Intent of these recommendations. It 
suggests that, in addition, further thought be given to the Incentives, 
or lack thereof, for labs to participate in Task Five as presently 
implemented. The panel also believes that the responsibility for success 
of Task Five is shared by OERI and the labs. While the labs need to live 
up to the spirit of the contract requirements, OERI trill need to take 
■trong central leadership if the potential in this ar>a is to be 
realized. 

Public and private ed ucation- With one exception, the team reports did 
not highlight the degree of services being provided to private 
education. The panel does not have enough information in this area of 
the program to comment knowledgeably. 

The panel believes that service to private echoole is of greater 
significance in some regions than others and that the interests of 
individual labs in it ahould be guided by their regional characteristics 
and governing boards. (Private school officials are among various types 
of educators the 1985 RFP recommended for consideration in board 
membe. -,hip.) The panel also believes the labs should be sensitive to the 
needs of intermediary organizations (i.e., through indirect service 
strategies) serving private education. Among these these organizations 
are those serving specific populations, such as the Association of 
Tribal-Controlled Schools. 

Regulatio n of the Labs- The panel is interested in the efficiency and 
productivity of the organizational and contractual relationships between 
the labs and OERI. Among the topics in vhich it is specifically 
interested are the following: 

- The degree of specificity in the lab contracts. The panel believes 
that the specificity of contracts emphasizes process at the expense of 
(1) a concept of overall lab role (2) substantive emphasis (3) visior. ef 
successful service and (A) indicators of success. 

- The panel believes the degree of reporting required by OERI mar be 
excessive. But given that, it is still possible, as suggested in' the 
discussion of clarity and vision of the labs' mission, the labs write 
too much in response to the OERI requirements. A greater balance in 
reporting requirements between the need for detailed accountability and 

•programmatic clarity needs to be sought. 

" The panel believes there is an inherent conflict between the greater 
degree of control and apecificity appropriately demanded in contracts 
and the autonomy lodged in the concept of regional labs. The panel is 
aware of the previous discussions about this aubject. It believes that 
the potential for use of cooperative agreementa in the future in the 
program is worthy of investigation. But for the moment, the panel would 
have to conclude that the ideal procurement mechaniam for use in the 
program does not exist. 
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Re.^i.-r.shlp Betvec-r. O ERI and non-OERI activities- The panel Is aware 
that labs differ greatly lr. the extent to which they have non-OERI 
funding. The Implications of this deserve attention. These Include: 

- Laboratories differ greatly in their entrepreneurial behavior as 
regards non-OERI funding. Some of the more mature ones have sizable 
proportions and amounts of such funding. On the one hand, this reflects 
organizational health and vitality. The panel Is concerned, however, 
that auch labs may be ahlftlng valued staff and other resources as 
necessary to compete for and conduct such services at the expense of the 
OERI-funded work. 

- The panel Is concerned that OERI may, unintentionally, be subsidizing 
labs to compete with other organizations for other OER: and non-OERI 
work. Labs may be afforded an advantage in competitions because of their 
relatively better-developed institutional capacity and accumulated 
abililty to tap other resources. The panel recognizes that the OERI 
contracts do not have that purpose and it is not OERI's intent to afford 
labs unfair advantage when competing for funds. Some means to ensure 
that this is not the case, however, or other resolution of this matter, 
may nonetheless be appropriate. 

- The panel is aware that newer labs without other sources of funding or 
cash reserves face unique needs regarding their cash flow. The panel 
believes that OERI should be responsive to such needs. 

Organizational maturity- An issue somewhat related to that of OERI and 
non-OERI activities is that of organizational maturity. The labs vary 
widely on this dimension. The older ones have been in existence f.r soce 
twenty years, the newer ones for two to three years. The following 
considerations, therefore, are worthy of attention regarding the labs' 
varying organizational maturity. 

- The panel believes the older labs might productively assist the 
younger ones on a selective basis in either governance, organization and 
managment, or programmatic areas. OERI staff have cited examples of how 
such intra-lab assistance has occurred. This is praiseworthy. The panel 
believes, however, that such ssslstsnce might be extended. Older labs 
might particularly take the lead in collaboration under Task Five. 

- With sll lsbs, but perhsps the older ones in psrticulsr, there is a 
need to obtain stsff to deal with new and unfolding educational areas 
(e.g.. higher order thinking skills), or retrain existing stsff to deal 
with them. The panel recognizes the need for stsbility in personnel 
sdministrstion of a lab and for some continuity in stsffing. On the 
other hand, stsbility and continuity will not always meet challenges in 
new areas. The panel has heard a concern that labs may be entering into 
fields where they do not hsve adequate stsff expertise, nor plsns to 
acquire that expertise. The psncl recommends that the labs' cspscity to 
adapt staffing to meet changes required in their mission snd clients' 
needs be examined. Specific factors to be examined include policies for 
providing new stsff. as required, and providing staff development for 
existing staff. Staff development can and should be s vitsl part of each 

O laboratory's overall personnel program* 
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Lab-specific matters- The panel will depart here from its focus on 
llll " comment on a few matters related to individual 

it :J t "ferred to are the newer ones. These comments are „ade 

in part because they relate to the specific labs and in part because 
the> exemplify the sorts of issues which may face' new institutions 
generally. They are thus relevant to the 1990 recompetition. 

IS!/!? 0 "" J" , thB North Central Re «ion«l Educational Laboratory reflect 
many strengths in the institution. They al.o reflect struggles the lab 

£ilS^..TSJ 1° * ChieVe ™«« delude marginally adequate 

£n.««. n ; ^ 1Ch K*I? DOV b,lDg •"bit.nti.lly renovated), turnover in 
management and cash flow problems related to Departmental parent 
procedures. (The lab did. however, have financial reserves available to 
it from a predecessor organization which prevented dire consequences 
from occurring due to the cash flow problems). "quences 

The present executive director of the lab asserts that the turnover in 
r^fr 6 ^ n °t -aversely affected it. Whatever the case in that 

lilllU 5rp? 8n ; ?! llBVe ! that £ ° r ' " ewer ln »itution with such a 
history OERI should continue to monitor the situation carefully. Two 
rather different monitoring postures are appropriate. One is to 
encourage and support appropropriate growth of the lab. The other, 
however, is to counsel moderation in aspirations, if necessary, so that 
the lab s progracmatic reach does not exceed its organizational and 
managerial grasp. 

The Southep t Educational Improvement Laboratory has had cash flow 

ISiiJS' '"I" W " e largely due t0 its netf status M a ^thout anv 
existing cash reserves to fall back upon. Difficulties in attaining a ' 
satisfactory payment mechanism experienced with the Department's 
contracts and finance offices have exacerbated the laboratory's cash 

tJat lllll'T* S 8 *" lnltlal Peri0d ' Th « P" el 18 P lMsed ^ note 

„ K fm BS 4PPear ° ear resolutlon - Th« P««l encourages close 
attention by OERI management to any similar situations in the future. 

The Regional Laboratory for the Northeast and Islands grew out of a 
pre-existing organization and is implementing a novel structure for 
providing service. State assistance centers have been established 
through existing organizations on a decentralized basis as a means to 
provide . substantial portion of the laboratory's services. The panel 
has heard arguments for and against the efficacy of this strategy. It 
has two observations te make in this regard. -cracegy. 

f£llJ b T , ? a ?!! ,l a 1 ** tha - tb ! Ub i§ 1 »P 1 «~«»S a strategy which 
was clearly included in its winning proposal in 1985. Any uncertainty. 

11**11 Un,M,t Vi t h th * n0VBl ch «"«.ri.tlc. of this .trJteU ought 
£r JiS 2V° J! tu ™ ed lnt0 Pr«*tur. Judgment, about ^success. 
The second observation is that, notwithstanding its contractual 
validity, enough experience has been gained with the strategy to warrant 

th.-fo- :f tlVlt V° l X?1 T J" Pr ° gr " B Md P'-P~f . ?S pane? 
therefore recommends that the lab be requester' to jointly convene such 
an activity with OERI during the first six months of the new progrlm 
yesr. The activity would examine the operation of the .Jr^gy to St.. 
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reactions to it fror. the field and possible future directions for it 
Representatives of various lab constituents and clients, plus others' 
vith the potential to contribute to the discussion, would be invited t 
participate. The panel would follow the course of this activity closel 



III. Recommendations Regardin g the Future of the Program 

The panel has a responsibility to nakc recommendatione covering the 
longer term of the lab program, as well as immediate considerations 
arising from the external review. A focal point for these longer tern 
considerations is policies relevant to the recompetition of the labs 
scheduled for 1990 and administration of the program thereafter. 

The panel makes the following recommendations at this time. They sre 
stimulated by discussion of this review. They are not, however 
necessarily derived from the review in every aspect. The panel members 

.15 f JJI°*Jk diVerS ! P e "> ectiv « to the task of school improvement 
and labs. The recommendations below in part reflect those perspectives. 

.Make the programmatic realities and the contractual req uirements 
compatible- ' 

The panel recognizes the need for OERI , or any funding agency, to be 
accountable for administration of its contracts. On the other hand, lab 
are engaged in work which, by definition, is client-driven to a 
considerable degree «nd thus changing in nature. The present contract 
requirements appear to Impose an unrealistic, degree of precision upon 
labs ir. stating in advance what they plan to do, at least for 
outyears , or those beyond the current and next ones. 

The panel has addressee this issue earl ier in this report. It recommend 
here that further study of desirable procurement procedures for new 
awards to xabs be explored as « r integral aspect of planning for the 
recompetition itself. 6 

Strengt hen Departmental program administration- 

The panel has not sought to examine the operations of the Department's 
Crants and Contracts Service (GCS) in support of the labs. It has no 
reason to believe that Departmental procurement regulations arc not 

lib re™"s rMOlvin8 C " h flow P robl «" ™* «low approval of other 

Just as the panel has counseled attention by labs to basic matters of 
ISSnr ^ •f ganl2ation ' the P™ 1 counsels OERI to take vhatever 
action Is possible to strengthen the contract administration of this 

Sr?;"!™ 11 ! dd i tlC T t0 ,, ^ ing • ddltl0Ml staffing in the relevant GCS 
Elf!. JtW ° nt • ddltitma3 "«P- * ia to hav. institutional 

liaieons (II s) gain maximum advantage of the powers which are lodged in 
their deeignation aa the "Contracting Officer's Technical 

SlIIIS t ;^?H (C ?T R) - Tht P ,ntl U * w "« «».t aome Government agencies 
delegate coneiderably more discretion to their program officers 
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having that designation than does GCS. :he panel strongly believes that 
appropriate training of II/s in their COTR role is desirable. 

The panel also notes that IL's are overburdened with administrative 
details. Such details are inappropriate for their role as senior staff 
and also hinder their addressing larger programmatic issues. Designation 
of junior staff to apprentice two or three Il/i is desirable. 
Apprentices could both assist IL f s with their administrative work and 
prepare to assume IL responsibilties when staff vacancies occur. Such 
staffing would also ensure that there would be adequate back-up 
knowledge about a lab and its region in the absence of the IL* 

Further examine the fiscal requirements of establishing and operating a 
lab- 

The panel has been struck by the vastly different digree of overall 
financial resources available to individual labs, particularly some 
r .jjparisonE between older ones and newer ones. Attention should be given 
to ways that labs' cash flow needs may be met. On the one hand, labs 
should not be required to endure financial hardship. On the other hand, 
they should not over-aggresively seek other sources of funds, possibly 
at the expense of OERI-funded activities or the labs 1 own sense of 
self-identity. The panel strongly recommends that consideration be given 
to including fees in any future contracts. Such fees are a reasonable 
way to develop reserves to meet unexpected need* and are in fact 
consistent with the contractual relationship between the department and 
the labs. 

Exacir.e t he "entrepreneurial" behavior of the labs- 

The panel believes the behavior of labs with large amounts of non-OERI 
funding is in fact a significant determinant of their overall 
institutional behavior. The panel recognizes that OERI is only 
accountable for conduct of work it funds, but it is not possible in 
every instance to understand performance of OERI-funded work without 
understanding the broader environment. 

The panel recommends that this "entrepreneurial" aspect of labs, 
specifically hov it affects OERI-funded work, be studied prior to the 
recompetltlon. Such a study would address patterns of seeking non- 
OERI-funded work by labs, the amounts and types conducted and the 
distribution of available resources within a lab between OERI-funded and 
non-OERI-funded work. 

Clarify the paperwork in the program- 

The combined requirements for paperwork imposed on the labs by the 
Department's contracts office and OERI are very large. Still, some labs 
seem to have gone beyond the bounds of what even OERI required in their 
submission of documents for this review. 

For the recompetltlon, documentation must be submitted in sufficient 
detail tc allow a valid and reliable review process to be conducted. For 
^ program administration, adequate documentation also needs to be 
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submitted. But as Indicated earlier, the panel believes the amount of 
documentation submitted In connection with this review may have actually 
obscured some larger vision of what the labs are doing. Clearer, more 
succinct documentation is possible in this program and should be 
encouraged. 

Study the implications of the indirect services ("with and through") 
strategy* 

The panel is impressed with the degree to which the labs have sought to 
implement this strategy. It has had major effects on the character and 
programs of many of the institutions. 

Still, there are many unanswered questions about the best way to 
implement this strategy and its implications. Among these questions are 
the most appropriate groups with whom to work "with and through", the 
effects of not serving some groups, the degree, if any, to which this 
strategy should be imbued in all the labs' work and appropriate ways to 
evaluate labs* performance working in this mode. 

A forthcoming field study appears to offer one opportunity to examine at 
least some of these issues. The strategy and its implications should be 
studied thoroughly prior to commencing the recompetition. 

Re-examine the assumptions underlying needs assessments 

The panel Is not convinced the extensive needs assessments are 
adequately serving their intended purpose of guiding programmatic 
planning vlthin the regions. On the one hand, there is too much data. Or. 
the other hand, the data do not always adequately track planning 
decisions and the allocation of services actually made. The panel 
recommends that the possibility of shorter needs assessments be 
explored, also that greater visibility be given to labs' unpublished 
bases for making decisions. 

More coordination is needed* 

The panel commends OERI and the labs for the efforts taken to coordinate 
activities among labs. But much more can and should be done. Within 
OERI, better coordination of the IL's activities can be accomplished. 
Better coordination of the lab program with other OERI-funded activities 
is particularly desirable. 

The labs can improve coordination among themselves and with other 
parties. As indicated earlier in this report, the panel does not believe 
that the labs are benefiting from iuch coordination of their activities 
to the extent they end their clients night. 

Further study of this issue is desirable. It should Include analysis of 
the successes and failures of the present coordination provision in the 
contracts (Task Five), but not be bound by the present contractual 
provisions. OERI should specifically study the potential for labs to 
further coordinate their work with the other Federally-funded assistance 
institutions listed in Table 1. 
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Study the future of services to the Pacific Baslo Region- 

The panel Is aware that the Northwest Regional Educational Laborator> is 
providing services to the Pacific Basin Region, This region covers a 
vast area, ranging from Hawaii in the ea»t to Pacific territories In the 
west. The area is strategically important, both educationally and 
otherwise. 

The area was designated a separate region in the 1985 recompetition of 
the laboratories. The Northwest lab was assigned to help the region 
develop full-service capacity to operate its own lab by the close of 
this contract period, as well as to provide services to it. 

The panel is not in a position to comment on the level or adequacy of 
services provided to the region by the Northwest lab, or the degree of 
progress made in preparing the region to have its own lab. Nonetheless, 
it strongly recommends that OERI study the future direction of lab 
services to this region prior to commencing the 1990 recompetition. 

More examination of the programs in the field is needed- 

The panel recommends that in the future, evaluations of whatever sort 
(monitors' visits, external reviews, studies, etc.). ahould seek to get 
more information from the people in the field with whom the labs work 
and serve. The panel recognizes the fiscal constraints on obtaining such 
information, i.e. funds for staff travel, reviews and contracted 
studies. It nonetheless encourages OERI to get as much "grass- roots' 1 
evidence as possible to ensuie that a clear and concise picture of how 
well the labs are working is obtained. 
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